# Combining fields
df['TITLE_CLEAN'] = df['TITLE_CLEAN'].fillna('unknown').astype(str).str.strip().str.lower()
df['SOFTWARE_SKILLS_NAME'] = df['SOFTWARE_SKILLS_NAME'].fillna('').astype(str).str.lower()
df['SPECIALIZED_SKILLS_NAME'] = df['SPECIALIZED_SKILLS_NAME'].fillna('').astype(str).str.lower()
# Combine text fields for TF-IDF
df['combined_text'] = df['TITLE_CLEAN'] + ' ' + df['SOFTWARE_SKILLS_NAME'] + ' ' + df['SPECIALIZED_SKILLS_NAME']Unsupervised Learning Model
Data Processing
Text Preprocessing: Combine Job Title and Skills into a Single Field for TF-IDF
- We combined the job title and skills into a single text field (combined_text) to create a richer, unified input for the TF-IDF vectorizer. This improves the quality of feature extraction by capturing more context about each job, enabling better clustering and analysis.
Unique Value Counts in Job Titles and Skill Fields
Unique values in 'TITLE_CLEAN': 27266
Unique values in 'SOFTWARE_SKILLS_NAME': 22456
Unique values in 'SPECIALIZED_SKILLS_NAME': 41462- It helps to assess the diversity and granularity of job titles and skill mentions before clustering or vectorization, which is important for understanding feature richness and potential noise in the dataset.
NLP + K Means Clustering
Text Vectorization and Feature Scaling for Clustering
#| eval: true
#| echo: false
# Vectorize
tfidf = TfidfVectorizer(max_features=1000, stop_words='english')
X_tfidf = tfidf.fit_transform(df['combined_text']).toarray()
# Scale
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X_tfidf) We performed text vectorization and feature scaling, which are essential preprocessing steps before clustering
Tfidf Vectorizer converts the cleaned job and skills text (combined_text) into a numeric matrix based on word importance (TF-IDF), enabling text-based clustering.
StandardScaler scales the TF-IDF features to have zero mean and unit variance, which is important because KMeans is sensitive to feature magnitudes.
KMeans Clustering and Evaluation with NAICS 6-Digit Labels
Adjusted Rand Index (NAICS_2022_6_NAME): 0.009
Normalized Mutual Info Score (NAICS_2022_6_NAME): 0.033Evaluate Clustering Using Multiple Reference Labels (NAICS, SOC, ONET)
#| eval: true
#| echo: false
from sklearn.metrics import adjusted_rand_score, normalized_mutual_info_score
reference_labels = ['NAICS_2022_6_NAME', 'SOC_2021_5_NAME', 'ONET_NAME']
results = []
for label in reference_labels:
df_eval = df[[label, 'cluster']].dropna()
ari = adjusted_rand_score(df_eval[label], df_eval['cluster'])
nmi = normalized_mutual_info_score(df_eval[label], df_eval['cluster'])
results.append({'Reference Label': label, 'ARI': ari, 'NMI': nmi})| Reference Label | ARI | NMI | |
|---|---|---|---|
| 0 | NAICS_2022_6_NAME | 0.0092 | 0.0331 |
| 1 | SOC_2021_5_NAME | 0.0000 | 0.0000 |
| 2 | ONET_NAME | 0.0000 | 0.0000 |
NAICS_2022_6_NAME has the highest agreement with clusters (though still very low), suggesting a slight alignment with industry-based classification.
SOC and ONET labels have zero alignment — meaning the clusters derived from TF-IDF features of job titles + skills do not correspond to occupation-based taxonomies.